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ABSTRACT 

Provided are hypotheses and evaluation forms to 
evaluate vocational-occupational-technical education (V.O.T.E.) 
programs whose purposes include the mainstreaming of handicapped 
students. Materials are geared toward testing the following five 
hypotheses: most occupational administrators can cite specific 
examples of formal or informal evaluations they conduct on a regular 
basis; most of the evaluations are used in formal or informal 
decision making; most of these decisions refer to some sort of 
resource allocation; most of these administrators perceive typical 
formal evaluation studies to be either irrelevant or unusable in 
their decision making; and most of these administrators perceive 
typical informal evaluations cited by their peers to be relevant and 
usable. Also provided is a glossary of terms and a test to pick out 
the most effective plan for each of five criteria* (SBH) 
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SUMMARY 



This publication contains hypotheses and evaluation forms that can help 
evaluators while assisting occupational educatiors to evaluate vocational- 
occupational-technical education (VOTE ) programs whose purposes include 
ft ma in st reaming handicapped students. 11 The hypotheses are strictly for the 
technical staff of researchers and evaluators. The forms are for busy 
administrators and the general public. This distinction is important. Managers 
are not to be distracted from decisions by hypotheses. 

The evaluation forms are ready to duplicate and use. Under no circumstances 
are the hypotheses to be given to individuals who don T t ask for them. Instead, 
simple explanations should be made available. 

As a result of using these forms, program evaluators should reach certain 
conclusions about criteria of evaluation REPORTS; 

1) Busy administrator? want FACTS that are inexpensive 
to gather. 

2) They want EXPLANATIONS that are directly linked to 
fact s* 

3) They want EVALUATIONS that pinpoint the trade-off 
benefits between GOALS (or plans) and COSTS (or 
budget) in terms of RESULTS (or performance). 

4) They want proof of PROGRESS from the STARTING 
POINT (or base) to the CURRENT STATUS (or most 
recent data). 

5) They want the evaluation to take care of technical 
terms (such as hypothesis) and tc REPORT facts, 
explanations, benefits, and progress in clear 

end concise statements. 

Any evaluator who doubts the above five criteria should check them. out. 
One way to do so would be to use the five evaluation forms contained in this 
publication. When divergence is documented, the evaluator can better respond 
to the individual needs of the administrator requesting an evaluation. 

Originally, these forms were developed for V.O.T.E. programs mainstreaming 
the handicapped. To appeal to a wider audience, the forms as herein reproduced 
have been generalized. 
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CONCLUSIONS 



INTRODUCTION 



The process of evaluating vocational-occupational-technical education 
(V.O.T.E.) can be operationalized either formally or informally. 

the plan presented herein can help administrators evaluate V.O.T.E. program 
whose purposes include ,, mainst reaming handicapped students. 11 

The evaluation tests out five hunches or hypotheses: 

1. Most occupational administrators can cite specific examples 
of formal or informal evaluations they conduct on a 
regular basis. 

2. Most of the evaluations are used in formal or informal 
decision making. 

3. Most of these decisions refer to some sort of resource 
allocation. 

4. Most of these administrators perceive typical formal 
evaluation studies to be either irrelevant or unusable 
in their decision making process. 

5. Most of these administrators perceive typical informal 
evaluations cited by their peers to be relevant and 
usable. 

The above five hypotheses have been stated ir simple non-technical language. 
The- exact hypotheses actually tested are given in the appropriate technical 
terminology on the following pages. 
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GLOSSARY 



V.O.T.E. 



Evaluate 



Evaluation as 
measurement 

Evaluation as 

professional 

judgment 

Evaluation as 
examination 

Evaluation as 
evolution 



Evaluation as 
public opinion 



Evaluation as 
accounting for 
tax dollars 

Formal 
evaluation 



Informal 
evaluation 

Hypothesis 

Testable 
hypothesis 



An acronyn, or abbreviation, for vocational-occupational-technical 
education* 

Determine the value of; for example, to evaluate V.O.T.E. programs 
means to determine the value of vocational-occupational-technical 
education program by measuring, judging, examining, evolving, 
publicizing, or accounting. 

The process of coming up with evidence that is countable, 
measurable, and relevant* 

The process of using expert human judges to obtain a quick, 
qualified*, and quantified assessment or ranking. 



The process of determining how well prespecified objectives were 
accomplished* 

A management information system which examines intermediate results 
to determine how ongoing activities can be improved and made more 
effective. 

A systematic needs assessment and follow-up to determine how well 
ongoing activities aire responding to the real needs and problems 
of a given coirmunity or target population. 

A tracking system that makes sure the cost of any activity is 
reasonable in relation to the anticipated results. 



Evaluation conducted with established design procedures, forms, 
formats, techniques, controls, measurements, guidelines, and 
criteria that can be used elsewhere for moderately different 
purposes. 

Evaluation conducted without an established design and without 
plans for replication elsewhere. 

An educated guess, hunch, or prediction. 

A guess, hunch, or prediction stated in such a way that evidence 
can be gathered to document a YES or NO answer. 
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Testable Hypothesis No. 1 



Hypothesis 

Most occupational administrators can cite specific examples of formal or 
informal evaluations they conduct on a regular basis. 

Testable Hypothesis 

In a sample of 20 occupational administrators, more than 10 can cite 
specific examples of formal or informal evaluations they conduct on a regular 
basis* 



INSIGHT 



When an occupational educator starts mainstreaming handicapped high school 
students into regular shops, laboratories, work stations, ar^d classroom, this 
process usually begins informally. Those handicapped individuals judged informally 
most likely to succeed are among the first applicants. Few formal labels are 
used. When these moderately handicapped students succeed, severely handicapped 
students are informally taken into the program. In the meantime, teachers and 
staff have had a chance to "tool up" informally. Students learn informally 
one-by-one how to live with, work with, and mature with peers exhibiting the broad 
spectrum of human potentials and possibilities. 

Such an informal introduction requires an informal evaluation. The program 
needs time to grow before being measured prematurely. An evaluation shapes any 
program, some way or another. An informal evaluation permits all concerned to 
put away preconceived fears and concerns about programs which "mainstream" 
handicapped and non-handicapped students in the same setting. Such a process 
makes all of us more humane. 

i 
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Tost Questions (Hypothesis No. 1) 

Directions: Answer the multiple choi.ce questions by checking the box in front 
of the answer you select. 

Answer the fill in requests that apply to you by writing your 
reply in the space provided. 



1. Are you an occupational administrator? 

□ YES I 1 NO 

2. What is your title as at? occupational administrator? 



3. Do you use formal evaluations as an occupational administrator? 

□ YES □ NO 

4. Cite one formal evaluation you have used as an occupational educator. 



5. Do you use informal evaluations as an occupational educator? 

□ YES I [ NO 

6. Cite one informal evaluation you have used as an occupational educator. 



7. Are most of the evaluations you use on £ regular basis as an occupational 
educator FORMAL or INFORMAL? 

f~| FORMAL □ INFORMAL 

8. Please explain your answer to question 7. 



9. Do you agree with the following statement, "Formal evaluations are assessments 
conducted with established procedures, forms, controls, measurement s» guidelines, 
and criteria that are scientifically acceptable." 

□ YES C3N0 

10. Do you agree with the following statement, "Most of the evaluations I conddct on 
a regular .basis, as an occupational administrator are formal evaluations." 

| I YES □ NO 

. ... -4-8 




Testable Hypothesis No. 2 



Hypothesis 

Most of the evaluations are used in formal or informal decision making. 
Testable Hypothesis 

In a minimum sample, of 10 evaluations identified in hypothesis no, l f 
more than 50% are used in either formal or informal decision making. 



INSIGHT 



Decision-making is more important than statistical analysis in many cases. 
For example, the following paragraph contains very little statistical back-up. 
However, many managers have used it for curriculum adaptation to help both 
handicapped and non-handicapped job applicants. Sometimes, the question to ask 
is THE OPINION WHY rather than merely the STATISTICAL HOW MANY, Here's an 
example: 

At least two out of three handicapped new employees who lose their jobs 
had adequate aptitude and ability. They seem to have lost their jobs because 
they couldn't get along with others on the job. They had conflicts with peers 
and supervisors. 

Decision-makers used this informal data to create a curriculum that zeroed 
in on: 

a) personality development 

(improve your assets) 

b) socialization 

(minimize your deficits by maximizing the assets of others) 

c) behavioral modification (you can CHANGE YOURSELF). 

P.S. The curriculum on job skills already existed. All it needed was UPDATING 
not creation. The new curriculum, on the contrary, had to be fie Id tested and 
revised several times. 
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Test Questions (Hypothesis No. 2) 

Directions: Answsr the multiple choice questions by checking the box in front 
of the answer you select. 

Answer the fill in requests that apply to you by writing your 
reply in the space provided. 

1. Does your job as an occupational educator involve you in decision making? 

| 1 YES □ NO 

2. What kinds of decisions do you make as an occupational educator? 

3. Do you make formal decisions as an occupational educator? 

□ YES □ NO 

4. Cite one foTmal decision you have made as an occupational educator. 



5. Do you make informal decisions as an occupational educator? 

| | YES £3 NO 

6. Cite one informal decision you have mad*?, as an occupational educator. 



7. Are most of the decisions you make on a regular b asis as an occupational 
educator FORMAL or INFORMAL? 

| 1 FORMAL INFORMAL 

8. Please explain your answer to question 7. 



9. Do you agree with the following statement, "A decision is only as good as the 
information or evaluation upon which it is based." 

| I YES 1 1 NO 

10. Do you agree with the following statement, lf Most of the evaluations I make are 
used in formal or informal decision making." , 



[□YES I I NO 



Testable Hypothesis No? 3 



Hypothesis 

Most of these decisions refer to some sort of resource allocation* 
Testable Hypothesis 

In a minimum of 10 decisions identified in hypothesis no. 2, more than 507. 
are used to allocate resources such as data, people, or things. 



INSIGHT 



Resource allocation directly relates to program objectives. Each 
person is different. Since each program has a different mix of in- 
dividual needs, the program resource allocations will vary. Evaluations 
must reflect this. There must be "enough room" for individual differences. 
Here's one example: 

Before the ma ins tr earning experiment, there was only one goal: (1) 
The graduate finds competitive employment. 

After mainstreaming, two more goals emerged. The length of time 
needed to complete the program became expandable. The product of the 
program included completers, who had learned enough to get a good job 
but not enough to graduate. The new goals are: (2) The graduate or 
completer finds placement in a community facility, such as a sheltered 
workshop; and (3) The graduate or completer returns home as an active 
and participating member of the family and community. The emphasis 
is on skill acquisition rather than on uniform completion time. 

Informal evaluation can give some idea of the effectiveness of 
"things hard to evaluate" $uch as: 

(a) Growth in responsibility 

(b) Integration into the regular schedule 

(c) Practical academic training 

(d) Activities of daily living 

(e) Vocational development 

(f) Maturity as a young adult 



Test Questions (Hypothesis No. 3) 

Directions: Answer the multiple choice questions by checking the box in front 
of the answer you select* 

Answer the fill in requests that apply to you by writing your 
reply in the space provided, ~t. 

1, Do you transmit DATA to your *t.aff on a regular basis? 

| I YES □ NO 

2. How do you decide whether or not to transmit data to your staff? 



3. Do you acquaint your staff with significant PEOPLE on a regular basis? 

□ YES □ MP 

4, How do you decide whether or not to acquaint your staff with significant 
people? 



5, Do you make available a variety of new THINGS to your staff on a regular 
basis? 

| 1 YES □ NO 

6. How do you decide whether or not to acquaint your staff with new things? 



7. With which of the following areas are most of your decisions concerned? 

[^2 DATA □ PEOPLE □ THINGS Q OTHER; specify 



Hypothesis 



Most of these administrators perceive typical formal evaluation studies 
to be either irrelevant or unusable in their decision making process. 

Testable Hypothesis 

In a sample of ''20 occupational administrators, more than 10 can cite 
specific examples of formal evaluation studies that they perceive as either 
irrelevant or unusable in their decision making process. 



Test Questions (Hypothesis No. 4) 

Directions: Answer the multiple choice questions by checking the box in front 
of the answer you select. 

Answer the fill in requests that apply to you by writing your 
reply in the space provided. 



1. Do you recognize the difference between FORMAL and INFORMAL evaluation 
studies? 

( | YES □ NO 

2. In a few words, what is the difference? 



3. Which type of evaluation do you PREFER? 

j | FORMAL □ INFORMAL 

4. Why? . 

5. Which type of evaluation do you find most RELEVANT, PERTINENT, and APPLICABLE 
to your decision making process as an occupational administrator? 

| | FORMAL □ INFORMAL 

6. Why? : 

7. Which type of evaluation do you find most USABLE, HELPFUL, and USEFUL in 
your decision making process as an occupational administrator? 

| 1 FORMAL □ INFORMAL 

8. Why? 

9. Which type of evaluation do you USE most often? 

1 | FORMAL INFORMAL 
10. Why? ; 
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Testable Hypothesis No. 5 



Hypothesis 

Most of these administrators perceive typical informal evaluations 
cited by their peers to be relevant and usable* 

Testable Hypothesis 

In a sample of 20 occupational educators, more than 10 report informal 
evaluation to be relevant and usable. 
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Test Questions (Hypothesis No, 5) 

Directions: Answer the multiple choice questions by checking the box in front 
of the answer you select. 

Answer the fill in request that apply to you by writing your 
reply in the space provided, 

1, Have you ever read or used an informal evaluation? 

| | YES □ NO 

2, In a few words, what was it all about? 



3, Have any of your peers ever cited or call an informal evaluation to 
your attention? 

| | YES □ NO 

4, How did it happen? 

5, How do you perceive most informal evaluation in general? 

\ | RELEVANT ]~] IRRELEVANT 

6, Why? p 

7, How do you perceive most informal evaluatin as far as use is concerned? 

( j USABLE □ UNUSABLE 

8, Why? ; 
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CONCLUSION 
«v 

Test out the following examples of plans for each criterion. Pick the one 
that seems most effective. Check your choice with the answer key. The answer 
key is found after the last criterion where it is printed upside down. 

Criterion 1 :' FACTS must be inexpensive to gather. 

Plan A . We will track down every graduate from this school, no matter how much 
it costs. 

Plan B . We will organize existing school records and sample 107. of those graduates 
whom we identify as "findable." 

. Criterion 2 ; EXPLANATIONS must be linked to facts. 

Plan C . We will tie all explanations to the 23 theoretical levels of employable 
personality intervening variables. 

Plan D« We will place our explanations in parallel columns alongside the data. . 

Criterion 3 : BENEFITS must accompany costs. 

Plan E. We will stress the intangible benefits of this study to show what 
MIGHT result ten years from now. 

Plan F . We will identify a COST for each BENEFIT whose existence we can document. 

Criterion 4 : PROGRESS must be documented* 

Plan G. We will stress a long list of things that still have to be done to 
achieve a perfect program. 

Plan H. We will document the growth between the pretest (at the beginning of 
the program) and the posttest (our most recent data). 

Criterion 5; EVALUATION must be readable. 

Plan I. We will dwell upon null hypotheses, Type I and Type II errors, exact 
chi-square values, t-tests, standard deviations, and the unexplained 
variances including co-variance. 

Plan J. We will provide a two page non- technical evaluation summary. Technical 
reports will be available, as required and as requested. 
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